There are many applications using speech recognition including applications for searching, command and control, spoken dialog systems, natural language understanding systems, and the like. Recently, Deep Neural Network (DNN) models being used in speech recognition applications have shown significant accuracy gains in many large vocabulary continuous speech recognition (LVCSR) tasks. However, DNN models use many more parameters and have a higher computation cost as compared to traditional systems (e.g. systems using a Gaussian Mixture Model (GMM)), which limits its deployment in many scenarios.